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After  the  Appraisal:  A  Systematic  Survey  of 
Process  Improvement,  its  Benefits,  and 
Factors  that  Influence  Success 


Abstract:  Very  little  published  evidence  exists  about  the  impact  of  the 
Capability  Maturity  Model®"’  (CMM)  or  CMM-based  appraisals  on  subsequent 
software  process  improvement  and  organizational  performance.  A  few 
credible  case  studies  do  exist,  but  it  is  uncertain  how  widely  their  results  apply. 
We  present  evidence  here  from  a  much  broader  cross  section  of  software 
organizations.  Our  results  suggest  that  process  maturity  does  indeed  pay  off 
in  better  product  quality,  ability  to  meet  schedule  commitrrients,  and  other 
indicators  of  organizational  performance.  The  vast  majority  of  survey 
respondents  also  report  that  their  appraisals  proved  to  be  highly  accurate  and 
useful  in  guiding  their  subsequent  process  improvement  efforts.  Not  all 
organizations  have  been  equally  successful,  however,  and  improvement  often 
takes  longer  and  costs  more  than  expected.  We  identify  several  factors,  most 
of  them  under  management  control,  that  distinguish  more  successful  from  less 
successful  organizations. 


1  Introduction 

1 .1  About  the  CMM  and  Process  Maturity 

The  Capability  Maturity  Model  (CMM)  [Paulk  93a,  Paulk  93b]  has  had  a  major  impact  on 
software  organizations  throughout  the  world.  Building  on  earlier  work  at  the  Software 
Engineering  Institute  (SEI)  [Humphrey  87],  the  CMM  is  used  as  a  reference  model  to  guide 
software  process  improvement  (SPI)  efforts  in  many  hundreds  of  software  organizations. 
Initially  adopted  by  defense  organizations  and  their  contractors,  it  is  now  used  in  organizations 
both  large  and  small  throughout  the  software  industry. 

As  seen  in  Figure  1-1,  the  CMM  describes  five  developmental  levels  of  software  process 
maturity.  At  the  initisl  level,  software  projects  depend  on  the  technical  skill  and  often  heroic 
efforts  of  specific  individuals.  They  proceed  in  an  ad  hoc  fashion,  from  one  issue  to  another. 
At  the  repeatable  level,  the  focus  is  on  establishing  effective  project  management  controls 
meant  to  enhance  product  quality,  and  to  improve  the  project’s  ability  to  set  and  meet 
reasonable  time  and  budget  commitments.  At  the  defined  level,  the  improvement  effort 
concentrates  on  developing  tailorable  software  processes  to  be  used  throughout  the  entire 
organization.  At  the  managed  level,  the  emphasis  is  on  monitoring  software  processes 
quantitatively  and  adjusting  them  to  better  meet  product  quality  goals.  Finally,  at  the 
optimizing  level,  quantitative  data  are  used  consistently  to  improve  the  organization’s 
processes  on  an  ongoing  basis. 


Capability  Maturity  Model,®"’  CMM,®"’  and  IDEAL®"’  are  service  marks  of  Carnegie  Mellon  University. 
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Source:  [Paulk  93a] 


Figure  1-1:  The  Capability  Maturity  Model 


Several  appraisal  methods  exist  that  are  meant  to  characterize  the  extent  to  which  a  software 
organization  has  established  processes  that  meet  the  criteria  established  by  the  CMM,  and  to 
guide  subsequent  process  improvement  [Paulk  92,  CBA  Project  94,  Whitney  94].^  Other, 
sometimes  competing,  approaches  exist,  but  the  influence  of  the  CMM  is  pervasive 
throughout  the  field  [Coallier  92,  Coallier  94,  Craigmyle  93,  Kuvaja  94,  Dorling  93,  Drouin  95]. 

1.2  The  Evidence  So  Far 

Very  little  published  evidence  exists  about  the  CMM  or  CMM-based  appraisals  that  goes 
beyond  individual  experience  or  strongly  stated  opinion.  However,  despite  doubt  from  some 
quarters  [Bollinger  91 ,  Bach  94,  Bach  95,  Jones  95],  what  little  publicly  available  evidence 
does  exist  is  quite  encouraging  to  proponents  of  the  CMM.  Several  case  studies  document 
well  conceived  and  implemented  process  improvement  efforts  which  returned  very  substantial 
business  value  to  their  organizations. 

''  •  Documentation  on  the  CBA  IPI  method  is  avaiiable  through  CBA  lead  assessor  training:  Members  of  the  CBA 
Project.  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA  IPI)  Lead  Assessor’s  Guide  v1.0. 
Pittsburgh,  PA:  Software  Engineering  institute,  Carnegie  Mellon  University,  May  1995. 
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A  recent  report  [Herbsleb  94]  reviews  process  improvement  efforts  in  1 3  organizations,  and 
shows  improvements  in  cycle  time,  defect  density,  and  productivity.  Benefit  to  cost  ratios 
presented  there  are  quite  impressive,  ranging  from  4:1  to  almost  9:1 .  Other  published  papers 
include  descriptions  of  process  improvement  efforts  at  Hughes  Aircraft  [Humphrey  91], 
Raytheon  [Dion  92,  Dion  93],  Schlumberger  [Wohlwend  93],  Texas  lnstruments[Benno  95], 
and  the  Air  Logistics  Center  at  Tinker  Air  Force  Base  [Lipke  92,  Butler  95].  Taken  together, 
these  case  studies  present  credible  evidence  about  what  can  happen  as  a  result  of 
CMM-based  SPI. 

1 .3  Limitations 

The  greatest  limitation  of  the  evidence  to  date  is  one  of  representativeness.  We  do  not  know 
how  representative  the  published  cases  are  of  the  experiences  of  CMM-adopters  in  general. 
Clearly,  not  every  software  organization  is  as  successful  as  those  that  choose  to  publicize  their 
experiences  widely.  Those  who  have  achieved  higher  maturity  may  be  more  anxious  to  tout 
their  accomplishments  than  are  those  who  are  experiencing  difficulty. 

We  need  to  proactively  seek  out  organizations  that  have  had  a  wide  variety  of  experience  in 
implementing  software  process  improvement.  We  need  to  learn  about  the  struggling  as  well 
as  the  outstanding  if  we  are  to  better  understand  the  characteristics  that  distinguish  between 
them. 

We  also  need  to  examine  the  experiences  of  different  types  of  software  organizations.  Early 
adopters  of  CMM-based  software  process  improvement  came  largely  from  defense 
contractors  and  military  organizations.  More  and  more  organizations  from  elsewhere  in  the 
software  industry  are  now  embarking  on  CMM-based  process  improvement  efforts.  We  need 
to  study  organizations  from  these  different  business  environments  in  order  to  understand  how 
broadly  applicable  the  CMM  is  as  a  model  for  SPI.  We  also  need  to  include  smaller 
organizations  in  order  to  objectively  evaluate  their  concerns  [Brodman  94]. 

1 .4  The  Survey 

This  survey  examines  appraisals  and  process  improvement  efforts  from  a  broad  cross-section 
of  software  organizations.  The  sample  includes  software  process  assessments  (SPAs)  that 
were  conducted  in  the  United  States  and  Canada  during  calendar  years  1992  and  1993  -  long 
enough  ago  for  genuine  change  to  have  taken  place  (at  least  one  year),  yet  recent  enough  to 
expect  accurate  recall  from  people  familiar  with  the  appraisals  and  their  aftermaths  (no  more 
than  three  years),  (See  Appendix  A  for  more  detail  about  the  sample.)  We  were  able  to  obtain 
information  allowing  us  to  contact  167  specific  individuals,  each  of  whom  was  in  a  good 
position  to  observe  the  aftermath  of  one  of  61  appraisals.  Using  an  intensive  schedule  of 
reminders  and  email,  we  received  138  completed  questionnaires,  which  is  83  percent  of  the 
total  number  sent.  They  represent  56  of  the  61  appraisals  (92  percent)  from  which  we 
sampled. 
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One  often  hears  that  process  champions  differ  substantially  from  managers  and  developers 
in  their  views  about  software  process  improvement.  Hence  we  designed  the  survey  sample 
to  allow  comparisons  among  people  whose  perspectives  might  be  expected  to  differ  as  a 
result  of  their  differing  roles.  The  data  reported  here  are  based  on  the  responses  of  individuals 
who  filled  one  of  three  roles  for  each  appraisal:  1)  the  project  level  software  manager  most 
knowledgeable  about  the  appraisal;  2)  the  most  knowledgeable  and  well-respected  senior 
developer  or  similar  technical  person  available;  and  3)  an  organizational  level  SEPG 
manager,  or  someone  with  equivalent  responsibilities,  if  such  a  person  existed. 

Interestingly  enough,  it  turns  out  that  there  are  not  characteristic,  systematic  differences 
among  the  respondents  who  fill  the  three  different  roles.  We  correlated  organizational  role 
with  the  respondents’  answers  to  all  of  the  other  survey  questions  described  in  this  report,  and 
found  only  two  statistically  significant  relationships  (p  <  .05  by  chi-square  criteria).  A  third 
approached  significance  (p  <  ,10).  With  enough  comparisons,  one  can  always  find  a  few 
apparently  significant  differences,  but  such  a  consistent  pattern  of  nonrelationship  is  highly 
unlikely  to  occur  by  chance.  (See  Appendix  A  for  more  detail.) 

The  overall  agreement  among  people  who  fill  different  organizational  roles  gives  us  more 
confidence  in  the  survey  results  than  we  would  have  had  if  we  relied  entirely  on  process 
champions.  Since  there  are  no  characteristic  role  differences,  we  have  combined  all 
respondents  for  the  analyses  presented  here.  The  larger  number  of  individual  respondents 
also  gives  us  more  confidence  in  the  overall  results  than  we  would  have  if  there  was  only  one 
respondent  per  organization. 

As  with  any  survey,  most  of  our  data  rely  on  the  self-reports  of  our  respondents,  and  we  do 
not  know  with  certainty  on  what  they  based  their  answers.  However,  all  that  must  be  true  for 
the  results  to  be  useful  is  that  if  we  ask  138  people  about,  e.g.,  the  ability  to  meet  schedules 
in  their  organizations,  those  who  say  that  it  is  “excellenf  or  “good”  usually  are  better  able  to 
meet  schedules  than  are  those  who  say  their  ability  is  “fair”  or  “poor.” 

Survey  data  are  necessary  if  we  wish  to  generalize  beyond  a  few  selected  cases. 
Furthermore,  there  is  evidence  that  people  in  fact  try  to  answer  survey  questions  honestly.  For 
example,  self  reported  and  appraised  maturity  levels  are  quite  consistent  in  this  survey  (see 
Section  3.1).  Our  respondents  also  describe  substantial  differences  in  process  improvement 
among  their  organizations,  and  those  descriptions  vary  among  each  other  in  understandable 
ways. 
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1 .5  The  Report 

Our  goals  for  the  survey  are  three-fold: 

1.  to  describe  what  typically  happens  to  process  improvement  efforts  after 
CMM-based  appraisals 

2.  to  understand  as  much  as  possible  about  why  some  improvement  efforts  are 
more  successful  than  others 

3.  to  learn  more  about  the  relationship  between  process  maturity  and 
organizational  performance 

The  remainder  of  this  report  contains  the  following  sections  and  appendices.  After  an  overall 
summary  of  the  survey  results  (Section  2),  we  present  several  sections  with  more  detailed 
results.  First,  we  discuss  new  evidence  about  the  impact  of  process  maturity  on  organizational 
performance  (Section  3).  This  is  followed  by  descriptions  of  the  appraisals  themselves 
(Section  4),  the  progress  of  process  improvement  since  the  appraisals  (Section  5),  and  a 
series  of  factors  that  distinguish  among  more  versus  less  successful  SPI  efforts  (Section  6). 
Finally,  we  present  our  conclusions  (Section  7). 

Appendix  A  describes  the  survey  sample  in  more  detail.  Appendix  B  provides  additional 
evidence  about  differences  due  to  organizational  size.  Appendix  C  contains  a  series  of 
figures  that  are  referenced  in  Section  6,  and  provides  additional  detail  about  factors  related  to 
varying  success  in  process  improvement.  The  survey  questionnaire  is  reproduced  in 
Appendix  D. 
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A  Summary  of  the  Survey  Results 


2.1  Impact  of  the  CMM 

A  number  of  conclusions  are  apparent  based  on  the  results  of  this  survey.  First  of  all,  we 
present  new  evidence,  based  on  a  much  broader  and  more  representative  sample  of  software 
organizations  than  heretofore  available,  that  process  improvement  does  pay  off  in  terms  of 
better  organizational  performance.  Our  respondents  from  higher  maturity  organizations  are 
considerably  more  likely  than  those  from  level  1  organizations  to  report  better  performance 
with  respect  to  product  quality,  staff  productivity,  ability  to  meet  schedule  commitments,  and 
their  own  staff  morale  and  job  satisfaction.  In  addition  they  generally  report  better 
performance  with  respect  to  customer  satisfaction  and  ability  to  meet  budget  commitments. 

The  basic  results  hold  up  for  organizations  from  different  sectors  of  the  software  industry, 
among  those  newer  to  the  CMM  as  well  as  those  from  defense  contractors  and  the  federal 
government.  The  results  also  appear  to  be  unaffected  by  the  size  of  the  organizations 
involved.  Organizations  with  relatively  few  software  employees  appear  to  benefit  from  higher 
process  maturity  just  as  do  larger  organizations. 

2.2  CMM-based  Appraisals 

On  the  whole,  our  survey  respondents  view  their  software  process  assessments  (SPAs)  as 
having  been  both  highly  accurate  and  useful  in  guiding  their  subsequent  process  improvement 
efforts.  This  is  true  in  light  of  their  actual  experience  in  the  one  to  three  years  following  the 
appraisals. 

The  respondents  report  that  the  appraisals  did  a  good  job  in  identifying  their  organizations 
strengths  as  well  as  their  weaknesses.  Most  report  that  their  organizations’  process 
improvement  efforts  have  been  largely  determined  by  the  results  of  their  appraisals.  Based 
on  their  experiences  following  the  appraisals,  a  large  majority  (over  80  percent)  believe  that 
the  CMM  has  provided  useful  “road  map”  direction  about  what  process  improvements  ought 
to  be  tackled  first.  Only  10  percent  now  think  that  their  appraisals  or  the  CMM  caused  them 
to  neglect  important  process  improvement  issues. 

There  are  difficulties  though.  Over  a  quarter  of  the  respondents  say  that  the  findings  and 
recommendations  raised  by  their  respective  appraisals  were  too  ambitious  to  achieve  in  a 
reasonable  time  period.  Large  numbers  of  the  respondents  said  that  they  needed  more 
assistance  and  guidance  about  how  to  achieve  tangible  improvement  in  the  areas  identified 
by  their  appraisals.  Knowing  what  to  improve  is  not  enough.  They  need  more  guidance  about 
how  to  go  about  making  the  improvements  actually  happen.  There  is  evidence  that  those  who 
reported  such  difficulties  in  fact  made  less  progress  in  their  subsequent  process  improvement 
efforts. 
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All  in  all,  however,  most  respondents  believe  that  the  money  and  effort  they  devoted  to  their 
appraisals  were  well  spent,  and  that  the  appraisals  had  a  substantial  positive  impact  on  their 
organizations. 


2.3  Progress  Since  the  Appraisals 

The  survey  respondents  report  that  a  substantial  amount  of  progress  has  taken  place  since 
their  appraisals  were  conducted.  Most  (56  percent)  of  the  respondents  report  that  their 
organizations  have  experienced  at  least  moderate  success  in  addressing  the  findings  and 
recommendations  that  were  raised  by  their  appraisals;  31  percent  report  substantial  success 
or  marked  success  throughout  their  organizations.  Only  14  percent  say  they  have  had  little  if 
any  appreciable  success  thus  far. 

The  vast  majority  of  the  respondents  report  having  followed  up  their  appraisals  with  action 
plans  and  process  action  teams  to  carry  out  those  plans.  Almost  three-fourths  said  that  their 
organizations  had  implemented  process  changes  in  demonstration  projects  or 
organization-wide  as  a  result  of  their  appraisals. 

There  also  is  evidence  that  the  appraisals  have  helped  establish  and  maintain  buy-in  and 
commitment  for  software  process  improvement.  Our  respondents  report  that  support  for 
process  improvement  has  improved  among  their  organizations’  management,  technical 
personnel,  and  appraisal  sponsors,  as  well  as  those  who  participated  directly  in  the  appraisals. 

Overall,  then,  the  evidence  from  our  survey  suggests  that  a  good  deal  of  progress  has  been 
made  since  the  appraisals.  There  is  very  little  evidence  indeed  that  the  appraisals  have  had 
a  negative  impact  on  the  progress  of  process  improvement.  Very  few  (4  percent)  of  our 
respondents  said  that  their  appraisals  have  been  counter-productive.  Contrary  to  some 
critics,  over  80  percent  of  the  respondents  said  that  their  organizations’  software  processes 
had  nof  become  more  bureaucratic  and  that  technical  creativity  had  not  been  stifled  since  their 
appraisals.  Indeed,  in  the  commercial  and  government  sectors,  there  is  evidence  that  more 
mature  organizations  have  fewer  paperwork  requirements  than  do  less  mature  organizations. 

Still,  we  detect  more  than  a  little  discouragement  about  the  pace  of  process  improvement. 
About  a  quarter  of  our  respondents  say  that  “nothing  much  has  changed”  since  the  appraisal. 
Almost  half  say  there  “has  been  a  lot  of  disillusionment  over  the  lack  of  improvement.”  Over 
40  percent  say  that  process  improvement  has  been  overcome  by  events  and  crises  and  that 
other  things  have  taken  priority.  Almost  three-quarters  tell  us  that  process  “improvement  has 
often  suffered  due  to  time  and  resource  limitations”;  over  three-quarters  say  that  process 
improvement  has  taken  longer  than  they  expected;  over  two-thirds  say  that  it  has  cost  more 
than  they  expected. 
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Such  difficulties  often  afflict  organizations  when  they  attempt  to  achieve  challenging  goals. 
Clearly,  though,  there  is  a  need  to  counter  unrealistic  expectations  about  process 
improvement  in  some  software  organizations.  Process  champions  and  managers  must  do  a 
better  job  of  managing  those  expectations  if  they  are  to  maintain  support  for  continuous 
process  improvement  over  the  long  haul. 

2.4  Prospects  for  Successful  Process  Improvement 

Our  respondents’  answers  to  several  questions  about  the  characteristics  of  their  organizations 
are  related  to  the  degree  of  success  they  attribute  to  their  process  improvement  efforts. 
Managers  can  take  a  number  of  actions  based  on  these  results. 

First  of  all,  managers  can  actively  monitor  the  progress  of  process  improvements  in  their 
software  organizations.  They  can  clearly  state  process  improvement  goals,  and  work  to 
ensure  that  adequate  resources  are  invested  in  their  process  improvement  efforts.  According 
to  our  survey  respondents,  organizations  that  fare  well  with  visible  management  and  support 
also  are  the  ones  most  likely  to  experience  success  in  addressing  the  improvements 
suggested  by  their  appraisals. 

Managers  can  also  have  at  least  some  control  over  the  ways  that  their  improvement  efforts 
are  staffed  and  compensated.  Staff  must  be  compensated  for  their  process  improvement 
efforts  as  part  of  their  normal  work  assignments.  Process  improvement  is  not  something  to 
be  accomplished  in  one’s  spare  time,  after  the  “real”  work  has  been  done.  People  involved  in 
process  improvement  should  be  well  respected  in  their  organizations.  Those  who  do  the 
technical  work  should  be  included  in  the  improvement  effort.  Process  improvement  is  not 
something  to  be  left  to  others.  Organizations  that  do  a  better  job  of  staffing  and  funding  the 
improvement  effort  are  also  the  ones  whose  post-appraisal  SPI  efforts  have  been  most 
successful  according  to  our  respondents. 

Our  data  suggest  a  number  of  factors  that  can  make  process  improvement  difficult  to  achieve. 
Aspects  of  organizational  culture  are  among  those  most  likely  to  inhibit  such  change.  When 
our  respondents  say  that  they  have  seen  excessive  turf  guarding  and  organizational  politics, 
they  also  report  less  success  in  addressing  the  findings  and  recommendations  that  were 
raised  in  their  appraisals.  Similar  results  exist  when  there  is  cynicism  and  discouragement  left 
over  from  previous  failures  or  when  the  technical  staff  tend  to  feel  that  SPI  gets  in  the  way  of 
the  “real”  work. 

Our  data  also  suggest  ways  in  which  the  research  and  development  community  can  contribute 
to  the  prospects  for  successful  software  process  improvement.  Those  survey  respondents 
who  say  that  the  recommendations  raised  by  their  appraisals  were  too  ambitious  are  also  less 
likely  to  report  successful  improvement  efforts  following  the  appraisals.  Similar  results  exist 
when  the  respondents  are  asked  about  the  need  for  more  guidance,  mentoring,  and 
assistance  in  implementing  the  improvements  suggested  by  the  appraisals.  We  need  to  learn 
more  about  howto  make  change  happen,  not  just  what  needs  to  be  improved. 
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3  Results  of  Software  Process  Improvement 


More  and  more,  the  process  improvement  community  has  been  seeking  objective  evidence 
about  the  effect  that  process  maturity  has  on  the  performance  of  software  organizations.  To 
that  end,  we  included  in  our  survey  a  series  of  questions  about  organizational  performance, 
and  correlated  the  answers  with  measures  of  process  maturity. 


3.1  Process  Maturity 

Although  we  know  the  appraised  maturity  levels  of  our  respondents’  organizations  (see 
Appendix  A),  we  thought  it  likely  that  some  of  them  would  achieve  higher  levels  in  the  one  to 
three  years  since  their  appraisals.  Thus  we  asked  them  to  estimate  their  current  maturity 
levels.  The  improvements  they  report  (Figure  3-1 )  are  consistent  with  what  one  would  expect 
based  on  their  appraised  maturity  levels  [Hayes  95].  Moreover,  there  is  relatively  little 
difference  between  the  appraised  and  reported  maturity  levels.  For  example,  77  percent  of 
those  whose  organizations  were  appraised  at  the  initial  level  say  that  they  still  are  at  level  1 ; 
79  percent  of  those  appraised  at  the  repeatable  level  say  that  their  organizations  still  are 
properly  classified  at  level  2. 


%  at  each 
Maturity 
Level 


Hi  appraised 
Hi  current 


Figure  3-1 :  Maturity  Level 


CMM-based  and  earlier  process  improvement  efforts  influenced  by  the  SEI  have  now  existed 
for  quite  some  time  among  government  contractors  [Humphrey  87].  Although  there  is  a 
growing  interest  in  software  process  improvement  among  commercial  companies,  such  efforts 
are  more  recent.  As  would  be  expected  given  their  shorter  exposure,  and  has  been  shown 
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Figure  3-2:  Maturity  Level  by  Sector 


elsewhere  [Zubrow  95],  the  commercial  companies  tend  to  have  less  mature  improvement 
efforts  than  the  government  contractors.  Indeed,  as  seen  in  Figure  3-2,  all  of  the  respondents 
who  say  they  have  achieved  level  three,  the  defined  level,  come  from  organizations  that  are 
government  contractors. 

As  seen  in  more  detail  in  Appendix  B,  our  sample  includes  organizations  of  varying  sizes. 
About  one-third  of  the  survey  respondents  report  coming  from  organizations  that  employ  over 
200  software  people.  Another  one-third  employ  70  or  fewer  such  individuals.  However,  there 
are  no  statistically  significant  or  consistent  differences  in  maturity  level  among  the  different 
sized  software  organizations. 

3.2  Impact  on  Organizational  Performance 

We  asked  the  survey  respondents  how  they  would  describe  their  organizations  with  respect 
to  six  performance  characteristics.  Two  of  them,  ability  to  meet  schedule  and  budget 
commitments,  address  process  predictability.  The  others  are  product  quality,  staff 
productivity,  staff  morale  /  job  satisfaction,  and  customer  satisfaction.  Performance  as 
described  by  our  respondents  on  all  six  characteristics  does  in  fact  differ  by  process  maturity. 

Each  line  in  Figure  3-3  shows  the  percentage  of  respondents  at  each  maturity  level  who 
characterize  performance  in  their  organizations  on  one  of  the  six  characteristics  as  “good”  or 
“excellent.”  As  seen  in  the  graphs,  higher  maturity  organizations  do  indeed  tend  to  perform 
better  than  do  those  who  remain  at  the  initial  level.  Five  of  the  six  correlations  with  maturity 
level  are  statistically  significant  (at  the  .05  level  according  to  chi-square  criteria).  The  sixth, 
ability  to  meet  budget  commitments,  approaches  statistical  significance.  There  is  an 
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unexplained  dip  in  reported  customer  satisfaction  at  the  repeatable  level.  However,  the  overall 
patterns  are  quite  clear.  Higher  process  maturity  does  appear  to  pay  off  in  better 
organizational  performance. 


Percent  of  “90od”  or  “excellenf 

Respondents  performance  on... 


+  -  ability  to  meet  schedules 
X  —  ability  to  meet  budget 


+  —  product  quality 
X  —  staff  productivity 


+  —  customer  satisfaction 
X  —  staff  morale 


Figure  3-3:  Impact  of  SPI  on  Organizational  Performance 
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Notice,  for  example,  that  80  percent  of  those  who  report  that  their  organizations  are  at  level  3 
say  that  their  ability  to  meet  schedule  is  good  or  exceilent.  Oniy  39  percent  of  those  who 
remain  at  the  initial  level  make  a  comparable  claim. 

Simiiarly,  notice  the  pattern  of  responses  about  product  quaiity.  Almost  one-fourth  of  those  at 
level  1  report  that  their  products  are  of  only  “fair”  or  “poor”  quality.  On  the  other  hand,  all  of 
the  respondents  who  report  that  their  organizations  have  achieved  levei  3  say  that  their 
product  quality  is  good  or  excellent.  Indeed  (not  shown  in  Figure  3-3),  almost  two-thirds  of 
those  who  claim  level  3  status  say  that  their  product  quality  is  excellent.  Only  eight  percent  of 
the  level  1  respondents  make  a  similar  claim. 

Those  respondents  claiming  higher  maturity  level  status  are  also  much  more  likely  to  report 
that  their  staff  morale  is  good  or  excellent.  As  seen  in  Figure  3-3,  fewer  than  a  quarter  (23 
percent)  of  those  at  the  initial  level  report  that  morale  is  good  (only  one  says  it  is  excellent)  in 
their  organizations;  indeed,  another  23  percent  (not  shown  in  the  figure)  say  that  their  morale 
level  is  poor. 

3.3  Impact  of  industrial  Sector  and  Organization  Size 

Recall  from  Figure  3-2  that  all  of  the  respondents  in  our  sample  who  claim  level  3  status  for 
their  organizations  are  federal  government  contractors.  However  the  characteristic 
relationships  between  process  maturity  and  organizational  performance  are  not  entirely  due 
to  the  level  3  organizations.  On  the  whole,  respondents  from  level  2  organizations  tend  to 
report  better  performance  than  those  from  level  1  organizations,  regardless  of  industrial 
sector. 

The  relationships  between  higher  maturity  level  and  three  of  the  performance  factors  (ability 
to  meet  schedule,  ability  to  meet  budget,  and  higher  staff  morale)  are  statistically  significant, 
even  excluding  the  government  contractors  from  the  analysis.  The  other  three  approach 
significance  by  chi-square  criteria.  All  six  relationships  are  similar  to  those  reported  in  Figure 
3-3.  Such  consistency  is  highly  unlikely  by  chance  alone. 

Similarly,  the  relationships  between  maturity  levei  and  organizational  performance  persist 
among  organizations  of  varying  size.  Again,  as  one  would  expect  with  small  sample  sizes  and 
relatively  few  higher  maturity  level  organizations,  the  individual  statistical  relationships  are  not 
always  significant.^  However  they  consistentiy  foilow  the  same  pattern:  higher  maturity  level 
is  associated  with  better  organizationai  performance.  See  Appendix  B  for  a  more  detailed 
analysis  of  results  for  small  organizations. 


Actually,  all  of  the  relationships  controlling  for  size  approach  statistical  significance  (at  the  .05  level  according 
to  chi-square  criteria).  Eight  of  eighteen  comparisons  are  in  fact  significant. 
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The  Appraisals 


4.1  Accuracy 

Our  survey  respondents  are  very  well  satisfied  that  their  appraisals  were  essentially  accurate 
(Figure  4-1).  After  up  to  three  years  of  experience  since  the  appraisals,  almost  all  of  them  still 
say  that  their  appraisals  were  generally  accurate  in  describing  their  organizations  major 
problems  with  software  process. 

Percent  of  major  problems 


Figure  4-1 :  Accuracy  in  Identifying  Major  Software  Problems 

While  software  process  appraisals  may  do  a  good  job  of  identifying  problems,  critics 
sometimes  complain  that  the  appraisals  do  not  give  proper  credit  for  existing  strengths.  If  true, 
such  a  situation  would  not  contribute  well  to  buy-in  for  improvement  efforts  based  on  appraisal 
results.  Our  respondents  tend  to  disagree  with  such  sentiments.  As  seen  in  Figure  4-2,  they 
are  less  satisfied  with  their  appraisals’  ability  to  recognize  strengths  than  weaknesses.  Over 
90  percent,  however,  report  that  the  appraisals  did  indeed  characterize  their  organizations’ 
strong  points  at  least  reasonably  well. 

Various  other  concerns  are  often  heard  about  the  reputed  inadequacies  of  process  appraisal 
methods.  Rather  few  of  our  respondents  share  such  concerns.  Eighteen  percent  do  say  that 
the  “results  were  too  dependent  on  the  expertise  and  judgment”  of  those  who  conducted  their 

Percent  of  major  strong  points 


Figure  4-2:  Accuracy  in  Identifying  Strong  Points 
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appraisals.  However,  very  few  (eight  percent)  say  that  the  wrong  people  were  chosen  to 
participate  in  the  appraisal  (be  interviewed,  fill  out  questionnaires,  etc.),  and  only  six  percent 
say  that  the  participants  were  not  fully  honest  with  the  appraisal  team.  Even  fewer  (four 
percent)  complain  that  there  was  insufficient  room  for  the  team  to  exercise  its  judgment  during 
the  appraisal. 

4.2  Actionability 

Beyond  being  accurate,  an  appraisal  must  provide  useful,  practical  information  on  the  basis 
of  which  managers  and  process  champions  can  successfully  take  action.  Based  on  their 
experience  over  the  past  one  to  three  years  since  their  respective  appraisals,  our  respondents 
generally  are  well  satisfied  that  the  appraisal  results  were  actionable.  As  seen  in  Figure  4-3, 
over  two-thirds  report  that  their  organizations’  software  process  improvement  efforts  were 
largely  determined  by  the  findings  and  recommendations  that  were  raised  in  the  appraisals. 
Fewer  than  1 0  percent  say  that  “very  little”  of  the  SPI  effort  was  based  on  the  appraisal  results. 

How  much  of  SPI  effort 
is  based  on  appraisal? 

almost  all  based  on  appraisal 
most  based  on  appraisal 
much  based  on  other  sources 
very  little  If  any  based  on  appraisal 
not  much  of  a  SPI  effort  to  speak  of 
don't  know 

0  10  20  30  40% 

Figure  4-3:  Impact  of  Appraisal  on  SPI  Effort 


As  seen  in  the  top  half  of  Figure  4-4,  over  80  percent  of  the  respondents  continue  to  believe 
that  the  “CMM  provides  valuable  direction  about  the  order  in  which  process  improvement 
should  be  made.”  The  value  of  the  CMM  as  a  “road  map”  notwithstanding,  almost  40  percent 
do  believe  that  the  CMM  fails  to  address  important  areas.  Still,  only  10  percent  think  that  the 
appraisal  and/or  the  CMM  caused  the  neglect  of  important  issues  facing  their  own 
organizations. 

The  data  in  the  bottom  half  of  Figure  4-4  do  present  a  somewhat  different  picture  though.  First 
of  all,  over  one-fourth  of  the  respondents  say  that  the  recommendations  resulting  from  their 
appraisals  proved  to  be  too  ambitious  to  accomplish  in  a  reasonable  time  period.  Perhaps  in 
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CMM  provides  valuable  "roadmap"  direction 
CMM  misses  important  areas 

neglected  other  important  issues 

Percent  who  agree  _ 

or  agree  strongly*  . 

need  guidance  about  how  to  improve 
need  mentoring  and  assistance 
recommendations  too  ambitious 


0  25  50  75  100% 


*  Each  bar  in  the  graph  summarizes  answers  to  a  separate  question,  so  the  percentages  do  not  total  100%. 

Figure  4-4:  Acting  on  the  Appraisal  Results 


a  related  vein,  large  numbers  of  the  respondents  agreed  that  they  need  more  assistance  and 
guidance  about  exactly  howto  implement  successful  process  improvement  programs. 

By  now,  much  is  known  about  how  to  appraise  process  maturity  and  identify  those  areas  most 
in  need  of  process  improvement.  More  and  more,  though,  we  hear  concerns  that  we  know  a 
lot  less  about  how  actually  to  achieve  tangible  improvement  in  the  areas  identified  by  the 
appraisals.  The  process  improvement  community  clearly  needs  to  address  such  issues  in 
more  detail.^ 


®  Current  work  in  this  area  includes  that  by  Peterson  94,  Basil!  92,  and  Weller  93,  as  well  as  new  work  by  Priscilla 
Fowler  and  others  at  the  SEI  on  the  use  of  detailed  “transition  packages.” 
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5  Progress  Since  the  Appraisals 

The  survey  respondents’  overall  judgments  are  quite  positive  about  the  progress  of  process 
improvement  since  their  appraisals.  Almost  three-quarters  of  them  agreed  to  the  rather 
strongly  worded  assertion  that: 

The  assessment  was  well  worth  the  money  and  effort  we  spent: 
it  had  a  major  positive  effect  on  the  organization. 

“Buy-in”  and  support  for  software  process  improvement  appears  to  have  improved  (Figure 
5-1).  We  asked  our  respondents  to  tell  us  about  support  for  SPI  both  prior  to  and  since  their 
appraisals.  Perhaps  not  surprisingly,  the  respondents  report  that  support  for  SPI  has 
increased  most  markedly  among  the  people  who  actually  participated  in  the  appraisals. 
However,  they  also  report  considerable  improvement  in  levels  of  commitment  among  their 
organizations’  management  and  technical  staffs,  and  even  among  the  appraisal  sponsors. 


participants  technical  staff  management  sponsors 

Figure  5-1 :  Support  for  Process  Improvement  Efforts 


The  respondents  also  report  that  their  organizations  have  made  good  progress  in  the  typical 
sequence  of  post-appraisal  activities  (Figure  5-2).  Close  to  100%  of  them  report  that  their 
organizations  created  action  plans  for  improvement  based  on  the  results  of  their  appraisals. 
Almost  90  percent  said  that  they  had  formed  process  action  teams  (PATs)  to  implement  the 
action  plans. 

Over  70  percent  said  that  their  organizations  had  piloted  process  changes  in  demonstration 
projects  as  a  result  of  their  appraisals,  and/or  had  implemented  changes  throughout  the 
organizations  based  on  the  appraisal  results.  Clearly,  these  organizations  have  progressed 
beyond  the  planning  stage  in  their  process  improvement  efforts.^ 

In  future  analyses,  we  will  focus  in  more  detail  on  the  nature  of  the  changes  they  implemented  and  on  how 
they  map  to  the  key  process  areas  (KPAs)  of  the  CMM. 
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Changes... 

in  pilot  projects  &  organization-wide 
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Figure  5-2:  Post-Appraisal  Activities 


Although  most  of  the  respondents  do  report  that  their  organizations  have  made  reasonable 
progress  in  taking  action  as  a  result  of  their  appraisals,  not  all  of  these  actions  have  been 
equally  effective.  We  asked  our  respondents  about  the  extent  of  success  their  organizations 
have  had  in  addressing  the  findings  and  recommendations  that  were  raised  as  a  result  of  their 
appraisals.  As  seen  Figure  5-3,  their  answers  vary  considerably.  As  we  will  see  in  Section  6, 
a  number  of  characteristics  of  the  organizations  and  their  SPI  efforts  distinguish  those  who 
have  had  tfie  most  success  from  those  who  have  been  less  successful. 


How  successfully  have  findings  and 
recommendations  been  addressed? 


Figure  5-3:  Success  in  Addressing  Appraisal  Results 
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While  there  is  substantial  variation  in  the  extent  of  process  improvement  reported  by  our 
respondents,  there  is  next  to  no  evidence  that  the  appraisals  have  negatively  affected  process 
improvement  in  the  organizations  where  they  were  conducted.  Very  few  (4  percent)  of  the 
respondents  said  that  the  appraisals  had  been  counter-productive  and  that  the  progress  of 
process  improvement  had  actually  worsened  since  their  appraisals. 

Contrary  to  some  of  the  more  loudly  voiced  criticisms  of  CMM-based  process  improvement, 
we  found  very  little  evidence  that  software  processes  have  become  more  rigid  and 
bureaucratic  or  that  it  has  become  harder  to  find  creative  solutions  to  difficult  technical 
problems.  A  large  majority  of  our  respondents  (84  percent)  disagreed  or  strongly  disagreed 
with  these  assertions.  Indeed,  respondents  from  more  mature  organizations  in  the 
commercial  and  governmental  sectors  reported  that  it  took  less  “paperwork  to  get  things 
approved”  than  did  respondents  from  less  mature  organizations.® 

That  said,  many  of  our  respondents  do  report  difficulties  in  maintaining  their  process 
improvement  efforts.  Over  a  quarter  of  them  (26  percent)  agree  that  “nothing  much  has 
changed”  since  the  appraisal,  almost  half  (49  percent)  say  that  there  “has  been  a  lot  of 
disillusionment  over  the  lack  of  improvement,”  and  42  percent  say  that  process  improvement 
has  been  overcome  by  events  and  crises  and  that  other  things  have  taken  priority.  Almost 
three-quarters  (72  percent)  report  that  process  “improvement  has  often  suffered  due  to  time 
and  resource  limitations.”  Over  three-quarters  (77  percent)  say  that  process  improvement  has 
taken  longer  than  they  expected,  and  over  two-thirds  (68  percent)  say  that  it  has  cost  more 
than  they  expected. 

The  reasons  for  these  difficulties  are  not  unique  to  SPI  efforts,  and  often  afflict  organizations 
when  they  attempt  to  achieve  challenging  goals.  Clearly,  though,  there  is  a  need  to  counter 
unrealistic  expectations  about  process  improvement  in  some  software  organizations.  Process 
champions  must  do  a  better  job  of  managing  those  expectations  if  they  wish  to  maintain 
long-term  support  for  continuous  process  improvement. 


®  The  relationship  between  higher  maturity  and  lessened  paperwork  does  not  hold  up  among  government 
contractors,  where  the  amount  of  paper  is  often  a  function  of  contractual  obligations. 
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6  Prospects  for  Successful  Process  Improvement 


Many  potential  barriers  exist  that  may  make  process  improvement  difficult  to  achieve. 
Similarly,  there  may  be  organizational  characteristics  that  help  make  some  process 
improvement  efforts  more  likely  to  succeed  than  others  [Fowler  90,  Miller  92,  Maher  94].  We 
included  several  questions  meant  to  capture  such  differences  among  the  organizations  in  our 
survey,  and  then  correlated  each  of  them  with  the  question  (summarized  in  Figure  5-3  on 
page  20)  about  how  successfully  the  organizations  had  addressed  the  findings  and 
recommendations  of  their  appraisals. 

6.1  Success  factors 

Successful  SPI  efforts  as  characterized  by  our  respondents  differ  from  less  successful  efforts 
in  several  ways.®  For  example,  as  seen  in  Figure  6-1 ,  a//  of  those  who  say  that  the  findings 
and  recommendations  raised  by  their  appraisals  have  been  addressed  with  marked  success 
throughout  their  organizations  also  report  that  their  managers  actively  monitor  the  progress  of 
process  improvement.  Such  management  commitment  is  considerably  less  common  in  the 
organizations  with  less  successful  improvement  efforts. 

senior  management  monitoring  of  SPI 
%  “substantial”  or  “moderate” 

100 

75 

50 

25 

0 

little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 

Figure  6-1:  Successful  SPI  and  Senior  Management  Monitoring  of  SPI 


Resource  issues  appear  to  be  quite  important.  For  example,  we  asked  our  respondents 
whether  there  “has  been  clear,  compensated  assignment  of  responsibilities  for  process 
improvemenf  in  their  organizations.  As  seen  in  Figure  6-2,  those  who  report  more  success 
in  their  improvement  efforts  also  are  much  more  likely  to  tell  us  that  there  is  such  explicit 
assignment  of  responsibility  for  SPI  in  their  software  organizations. 

For  reasons  of  space,  only  a  few  characteristic  figures  are  included  in  Section  6.  Figures  for  the  other  good 
predictors  of  SPI  success  (p  <  .05  by  chi-square  criteria)  are  reproduced  in  Appendix  C 
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compensated  SPI  responsibilities 
%  “substantial”  or  “moderate” 

100-1 - 


little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 

Figure  6-2:  Successful  SPI  and  Compensated  SPI  Responsibilities 


Four  other  factors  are  comparably  associated  with  successful  SPI  efforts.  Those  who  claim 
greater  success  in  addressing  the  improvements  suggested  by  their  appraisals  also  are  more 
likely  to  report  that 

•  the  people  involved  in  process  improvement  have  been  well  respected  in 
their  software  organizations  (Figure  C-5) 

•  there  has  been  more  involvement  of  technical  staff  in  the  SPI  effort 
(Figure  C-6) 

•  the  amount  of  staff  time  and  resources  dedicated  to  process  improvement 
has  been  good  or  excellent  since  their  appraisals  (Figure  C-7) 

•  process  improvement  goals  are  clearly  stated  and  well  understood  in  their 
organizations  (Figure  C-8) 

6.2  Barriers 

Not  surprisingly,  we  also  have  evidence  about  barriers  that  can  inhibit  successful  software 
process  improvement.  As  can  be  seen  in  Figure  6-3,  excessive  “organizational  politics”  seem 
to  be  particularly  damaging.  Barely  a  quarter  of  our  respondents  who  claim  marked  success 
for  their  organizations’  SPI  efforts  also  report  an  inordinate  amount  of  organizational  politics. 
Three-quarters  of  those  who  report  little  if  any  success  in  addressing  the  findings  and 
recommendations  of  their  appraisals  say  that  organizational  politics  are  commonplace. 

Three  other  factors  are  comparably  associated  with  less  successful  SPI  efforts.  Similar  results 
exist  for 

•  “turf  guarding”  (Figure  C-9) 

•  discouragement  and  cynicism  from  previous  experience  (Figure  C-10) 

•  the  feeling  among  the  technical  staff  that  process  improvement  gets  in  the 
way  of  their  “real”  work  (Figure  C-1 1 ) 
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organizational  politics 
%  “substantial”  or  “moderate” 


100- 


little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 


Figure  6-3:  Successful  SPI  and  Organizational  Politics 


Figure  6-4  shows  the  relationship  between  reported  success  of  SPI  and  our  respondents’ 
answers  to  the  question  about  the  scope  of  the  findings  and  recommendations  raised  in  their 
appraisals.  Those  with  less  successful  process  improvement  efforts  are  also  more  likely  to 
say  that  their  appraisals’  recommendations  were  too  ambitious.  Similar  results  exist  for  the 
two  questions  about  need  for  more  guidance,  mentoring,  and  assistance  in  implementing  the 
improvements  suggested  by  the  assessments  (Figures  C-12,  C-13,  and  C-14).^ 


recommendations  too  ambitious 
%  “agree”  or  “strongly  agree” 


Success  addressing  findings  /  recommendations 


Figure  6-4:  Successful  SPI  and  Recommendations  Too  Ambitious 


The  relationship  in  Figure  6-4  is  not  quite  significant  by  chi-square  criteria,  and  the  data  in  Figure  C-13  are 
rather  “noisy”  (the  more  characteristic  differences  exist  for  the  respondents  who  “strongly”  agree  or  disagree 
that  there  is  a  need  for  mentoring  and  assistance  in  their  organizations).  However,  all  three  relationships  are 
very  unlikely  to  occur  together  by  chance  alone.  See  the  discussion  of  Figure  C-1 4  on  page  48  for  more  detail. 
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Four  other  inhibiting  factors  are  moderately  related  to  our  measure  of  overall  SPI  success 
since  the  assessments.  The  correlations  are  only  marginally  significant  statistically  (p  >  .05 
by  chi-square  criteria).  However  they  are  of  sufficient  interest  to  mention  here,  and  quite 
possibly  worthy  of  further  consideration  in  future  work.  They  are 

•  turnover  in  key  senior  management 

•  the  need  for  paperwork  to  get  things  approved  in  the  organization 

•  decreasing  demand  for  the  organization’s  products  or  services 

•  major  reorganization (s)  or  staff  down-sizing 

6.3  What  Doesn’t  Seem  to  Matter? 

Not  all  of  the  potential  success  factors  or  inhibitors  that  we  examined  proved  to  be  good 
predictors  of  SPI  success.  How  the  respondents  answered  these  questions  is  unrelated  to 
their  answers  to  the  question  about  overall  success  in  addressing  the  findings  and 
recommendations  raised  by  their  assessments 

•  whether  or  not  software  organizations  provide  special,  tangible  incentives 
and  rewards  for  successful  SPI 

•  turnover  among  middle  management  and  technical  staff 

•  management  willingness  to  take  risk 

•  the  extent  to  which  management  understands  the  “issues  faced  by 
practitioners” 

It  is  important,  however,  to  interpret  these  results  in  the  context  of  our  results  about  factors 
that  are  in  fact  related  to  SPI  success.  For  example,  while  special  rewards  for  successful  SPI 
do  not  appear  to  have  a  consistent  effect,  making  sure  that  those  responsible  for  process 
improvement  are  properly  compensated  as  part  of  their  regular  work  efforts  is  important 
(Figure  6-2).  Detailed  understanding  of  the  technical  work  by  senior  management  (and/or 
micro-management)  does  not  have  a  consistent  impact  on  the  success  or  failure  of  the 
software  process  improvement  efforts  described  by  our  respondents.  However,  senior 
management  oversight  is  in  fact  quite  important  (Figure  6-1). 
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7  Conclusion 


We  began  this  report  by  enumerating  three  goals  for  the  survey.  Here,  we  briefly  discuss  the 
extent  to  which  we  were  able  to  meet  them,  and  then  identify  some  important  remaining  issues 
for  future  work. 

7.1  Discussion 

Goal  1 :  Describe  what  typically  happens  to  process  improvement  efforts  after  CMM-based 
appraisals. 

This  survey  provides  a  much  needed  description  of  the  experiences  of  software  organizations 
that  have  based  their  process  improvement  efforts  on  the  CMM  and  CMM-based  appraisals. 
Compared  to  previous  work,  the  survey  is  much  more  representative  of  the  CMM-based  SPI 
community.  It  is  drawn  from  commercial  and  government  organizations  as  well  as 
government  contractors,  and  it  includes  organizations  that  vary  considerably  in  size.  It 
includes  both  more  and  less  successful  SPI  efforts,  and  it  spans  several  maturity  levels. 

Our  respondents  are  drawn  equally  from  senior  technical  staff  and  software  managers,  as  well 
as  SEPGers  and  other  process  improvement  champions.  Our  results  are  not  dependent 
solely  on  people  who  are  personally  invested  in  the  CMM.  Indeed,  there  are  no  systematic 
differences  among  the  three  groups  from  which  we  sampled. 

We  sent  questionnaires  to  every  person  we  could  identify  who  was  in  a  good  position  to  see 
what  happened  in  the  one  to  three  years  following  an  SEI  software  process  assessment.  We 
followed  up  aggressively  to  get  a  good  (83  percent)  rate  of  return  and  avoided  self-selection 
problems. 

Goal  2:  Understand  as  much  as  possible  about  why  some  improvement  efforts  are  more 
successful  than  others. 

We  found  a  number  of  attributes  of  software  organizations,  their  SPI  efforts,  and 
organizational  cultures  that  are  strongly  related  to  success  in  process  improvement.  Many  of 
these  are  under  direct  management  control.  Some  of  the  cultural  factors  may  be  difficult  to 
overcome,  but  good  managers  deal  with  such  issues  every  day. 

Goal  3:  Learn  more  about  the  relationship  between  process  maturity  and  organizational 
performance. 

We  found  a  number  of  important  differences  in  performance  between  more  mature  and  less 
mature  organizations.  These  results  are  largely  consistent  with  the  case  studies  in  the 
literature  and  with  our  previous  work. 
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7.2  Remaining  issues 

No  single  study  can  hope  to  test  all  of  the  important  ideas  and  claims  about  the  CMM  and  what 
happens  as  organizations  implement  SPI  efforts  based  on  it.  The  CMM  is  a  complex 
reference  model,  and  the  effects  of  organizational  change  are  difficult  to  measure  and  difficult 
to  trace  back  to  particular  causes. 

We  currently  are  working  on  a  coordinated  series  of  studies  to  address  in  more  detail  the 
content  of  the  CMM  and  the  consequences  of  adopting  it.  One  line  of  work  we  have  recently 
undertaken  is  moving  toward  the  establishment  of  a  series  of  “collaboratories”  with  a  small 
number  of  software-dependent  organizations.  The  approach  is  within  the  spirit  of  the  Software 
Engineering  Laboratory  pioneered  by  Basili  and  his  associates  [Basili  92],  but  it  focuses 
primarily  on  software  process  improvement  and  on  how  to  determine  and  achieve  business 
results.  In  an  attempt  to  provide  more  actionable  guidance  for  SPI,  other  studies  will  focus  in 
depth  about  factors  that  affect  the  success  of  improvement  efforts  within  particular  key 
process  areas. 
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Appendix  A  The  Survey  Sample 


A.1  The  Appraisals 

The  sample  for  this  survey  was  drawn  from  the  Process  Appraisal  Information  System  (PAIS) 
database  maintained  at  the  Software  Engineering  Institute.  It  includes  software  process 
assessments  (SPAs)  that  were  conducted  in  the  United  States  and  Canada  during  calendar 
years  1992  and  1993  -  long  enough  ago  for  genuine  change  to  have  taken  place,  yet  recent 
enough  to  expect  accurate  recall  from  people  familiar  with  the  appraisals  and  their  aftermaths. 
Appraisals  conducted  outside  of  North  America  were  excluded  to  avoid  undue  administrative 
costs. 

The  sample  was  created  in  September  of  1994.  At  that  time,  155  SPAs  in  the  PAIS  database 
met  our  time  and  geographic  selection  criteria.  We  were  able  to  obtain  information  allowing 
us  to  contact  specific  individuals  for  the  survey  from  61  appraisals  -  slightly  less  than  40 
percent  of  those  eligible. 

Not  all  of  our  original  points  of  contact  from  the  database  were  equally  accessible  or 
accommodating,  and  we  did  have  difficulty  in  finding  individual  contact  information.  However, 
there  is  no  a  priori  reason  to  expect  any  bias  in  the  sample  of  61  appraisals  as  compared  to 
the  155  eligible  appraisals. 

In  particular,  the  appraisals  in  the  sample  do  not  appear  to  be  self-selected.  The  appraisals 
for  which  we  were  unable  to  obtain  individual  contact  information  were  conducted  at  a  variety 
of  software  organizations,  including  some  well  known  for  their  successful  improvement  efforts. 
As  seen  in  the  main  body  of  this  report,  the  survey  respondents  reported  widely  varying 
degrees  of  success  in  the  process  improvement  efforts  subsequent  to  their  appraisals.  Even 
if  the  organizations  we  included  are  somehow  more  successful  than  the  others  in  their  process 
improvement  efforts,  there  would  have  to  be  very  substantial  bias  in  the  sample  to  invalidate 
our  basic  results,  especially  those  comparing  success  among  different  types  of  organizations. 

The  present  survey  contains  a  series  of  questions  that  we  had  used  in  an  earlier  survey  based 
on  a  very  different  sample  [Deephouse  95].  Results  from  both  surveys  are  very  similar  about 
the  extent  of  process  improvement  following  comparably  recent  appraisals.  (See  the 
discussion  of  Figure  5-2  for  further  details.)  Such  consistency  improves  our  confidence  in  both 
samples. 

Of  course,  we  have  no  way  of  knowing  the  extent  to  which  the  PAIS  database  itself  is  fully 
representative  of  all  CMM-based  appraisals.  Although  the  database  has  grown  considerably 
in  recent  years,  it  undoubtedly  is  incomplete.  We  expect  that  appraisals  will  be  logged  in  PAIS 
much  more  regularly  in  the  future  given  the  requirements  for  authorization  of  lead  assessors 
under  the  program  for  CMM-Based  Appraisals  for  Internal  Process  Improvement  (CBA  IPIs) . 
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As  will  be  seen  momentarily,  the  individual  respondents  to  our  surveys  have  been  very 
accommodating.  We  have  achieved  very  good  return  rates,  so  we  can  be  confident  that  our 
results  are  not  biased  by  self-selection  effects  among  the  respondents.  However  it  bears 
repeating  here  that  the  quality  of  our  analyses  depends  ultimately  on  the  degree  of 
cooperation  we  receive  from  the  CMM-based  process  improvement  community. 

A.2  The  Respondents 

People  who  fill  different  roles  in  an  organization  can  sometimes  have  differing  perspectives 
about  the  same  events.  For  example,  one  often  hears  that  the  views  of  software  developers 
and  managers  typically  are  quite  unlike  those  of  the  champions  of  software  process 
improvement  in  their  own  organizations.  Similarly,  managers  and  technical  people  often  are 
seen  as  being  widely  divided.  Hence  we  designed  the  survey  sample  to  allow  comparisons 
among  people  whose  perspectives  might  be  expected  to  differ  as  a  result  of  their  differing 
roles 

We  asked  our  original  database  points  of  contact  for  each  appraisal  to  nominate  individuals  to 
fill  four  roles:  1 )  the  project  level  software  manager  most  knowledgeable  about  the  appraisal; 
2)  the  most  knowledgeable  and  well-respected  senior  developer  or  similar  technical  person 
available;  3)  an  organizational  level  SEPG  manager,  or  someone  with  equivalent 
responsibilities,  if  such  a  person  existed;  and  4)  the  senior  manager  who  was  the  appraisal 
sponsor,  or  his/her  replacement.®  Excluding  the  appraisal  sponsors,  we  obtained  contact 
information  for  167  individuals  representing  the  61  appraisals.® 

Interestingly  enough,  it  turns  out  that  there  are  not  characteristic,  systematic  differences 
among  the  respondents  who  fill  the  three  different  roles.  We  correlated  organizational  role 
with  the  respondents’  answers  to  all  of  the  other  survey  questions  described  in  this  report,  and 
found  only  two  statistically  significant  relationships  (p  <  .05  by  chi-square  criteria).  A  third 
approached  significance  (p  <  .10).  With  enough  comparisons,  one  can  always  find  a  few 
apparently  significant  differences,  but  such  a  consistent  pattern  of  non  relationship  is  highly 
unlikely  to  occur  by  chance. 

If  anything,  the  SEPGers  actually  tend  to  be  slightly  less  satisfied  than  the  others  about  the 
progress  of  software  process  improvement  since  their  appraisals.  However  the  differences 
are  minor,  and  exist  by  the  most  generous  criteria  in  fewer  than  one  fourth  of  the  role 
comparisons  we  made.  More  often,  we  simply  find  no  differences  attributable  to  role. 

®  We  chose  not  to  include  the  appraisal  sponsors  in  the  current  survey.  Many  of  the  current  questions  are  of  a 
more  technical  than  managerial  nature,  and  we  wanted  to  tailor  a  more  focused  set  of  questions  for  the 
sponsors  based  on  what  we  would  learn  from  the  broader  survey. 

®  There  are  not  three  people  for  all  of  the  appraisals.  Not  all  three  roles  (senior  technical,  project  management, 
and/or  process  champion)  were  always  filled.  In  a  few  Instances  the  same  person  held  more  than  one  role. 
More  than  one  person  shared  a  single  role  in  a  few  other  instances. 

There  was  no  requirement  that  any  of  these  people  were  on  the  appraisal  teams,  nor  even  that  they  personally 
participated  in  the  appraisals.  However,  they  all  were  required  to  be  familiar  with  their  respective  appraisals, 
and  with  the  progress  of  software  process  improvement  in  their  organizations  since  the  appraisals. 
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The  overall  agreement  among  people  who  fill  different  organizational  roles  gives  us  more 
confidence  in  the  survey  results  than  we  would  have  had  if  we  relied  entirely  on  process 
champions.  Since  there  are  no  characteristic  role  differences,  we  have  combined  all 
respondents  for  the  analyses  presented  in  this  report.  The  larger  number  of  individual 
respondents  also  gives  us  more  confidence  in  the  overall  results  than  we  would  have  if  there 
was  only  one  respondent  per  organization. 

A.3  Response  Rates 

We  sent  questionnaires  by  mail  to  the  167  individuals  in  our  sample  during  the  period  from 
November  1994  through  April  1995.  Follow-up  reminders  and  replacement  questionnaires 
were  sent  as  necessary.  The  analyses  in  this  report  are  based  on  138  completed 
questionnaires,  which  is  83  percent  of  the  total  number  sent.^*^  They  represent  56  of  the  61 
appraisals  (92  percent)  from  which  we  sampled.  We  can  be  quite  confident  that  there  is  little 
if  any  self-selection  bias  among  our  respondents.^^ 

A.4  About  the  Appraised  Organizations 

Our  survey  respondents  represent  a  variety  of  software  organizations.  The  largest  single 
proportion  (37  percent)  are  from  organizations  that  do  contract  work  for  the  federal 
government.  Another  22  percent  are  from  the  federal  government  and  U.S.  military  services. 
These  figures  are  not  surprising  given  the  long-standing  experience  with  software  process 
improvement  among  such  organizations.  As  expected  from  recent  updates  from  the  PAIS 
database  [Zubrow  95],  firms  selling  products  in  the  commercial  market  are  the  second  largest 
category  (36  percent)  of  software  organization  represented  by  our  respondents.  Another  5 
percent  fall  into  the  “other”  category. 

The  organizations  represented  in  our  sample  vary  considerably  in  size.  Approximately  one- 
third  of  the  survey  respondents  say  they  come  from  organizations  that  have  200  or  more 
software  employees.  Another  third  come  from  organizations  that  employ  70  or  fewer  people 
who  are  primarily  engaged  In  software. 

Firms  selling  products  in  the  commercial  market  are  smaller  than  those  in  the  military  and 
federal  government;  43  percent  of  the  commercial  organizations  have  70  or  fewer  software 
employees,  as  opposed  to  only  14  percent  of  the  government  organizations.  The  government 
contractors  vary  more  in  size;  40  percent  have  200  or  more  software  employees,  while  34 
percent  have  70  or  fewer. 


10  We  received  two  additional  questionnaires  after  the  analyses  were  completed. 

11  Mail  surveys  tend  to  suffer  from  disturbingly  low  response  rates.  Worse,  there  is  evidence  that  people  who  fail 
to  return  completed  questionnaires  without  additional  prodding  typically  differ  in  important  ways  from  those 
who  do  respond  right  away.  However,  as  our  experience  attests,  it  is  possible  to  attain  high  response  rates  to 
mail  surveys  when  proper  attention  is  paid  to  implementation  issues. 


CMU/SEI-95-TR-009 


39 


Most  (83  percent)  of  the  respondents  report  that  their  organizations  have  software  engineering 
process  groups  (SEPGs)  or  other  units  that  perform  similar  functions;  46  percent  report 
having  SEPGs  in  their  parent  (e.g.,  corporate  level)  organizations.  The  overall  effort  devoted 
to  software  process  improvement  varies  considerably.  One-fourth  say  that  they  employ  3  or 
fewer  full-time-equivalent  (FTE)  people  who  have  specifically  assigned  responsibilities  for 
process  improvement  and/or  quality  management.  Half  employ  5  or  fewer  such  people 
FTE,  but  another  one-fourth  employ  12  or  more. 

As  expected,  the  survey  respondents  are  pretty  much  evenly  distributed  among  the  roles  that 
we  sampled:  31  percent  are  SEPGers  and  other  process  champions;  34  percent  each  are 
software  managers  or  senior  technical  people  respectively.  One  person  filled  both  the 
management  and  SEPG  roles  concurrently. 

The  respondents  have  a  considerable  amount  of  software  experience.  Half  of  them  have 
worked  on  software  for  1 6  years  or  more;  a  quarter  of  them  have  worked  in  the  field  for  22  or 
more  years.  All  but  the  least  experienced  1 0  percent  of  our  respondents  have  worked  on 
software  for  10  years  or  more. 


FTE  is  defined  in  the  survey  as  “full  timers  plus  the  hours  worked  by  part  timers  and  consultants.” 
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Appendix  B  Differences  Due  to  Organizational  Size 

Whether  or  not  CMM-based  process  improvement  scales  down  well  to  smaller  organizations 
has  been  the  subject  of  a  continuing  and  sometimes  heated  debate  in  the  process 
improvement  community  [Brodman  94].  Our  survey  contributes  at  least  some  limited, 
objective  data  to  the  dialog. 

First  of  all,  (as  mentioned  in  Section  3.1)  organizational  size  per  se  is  not  related  to  process 
maturity.  There  are  no  statistically  significant  or  consistent  differences  in  maturity  level  among 
the  different  sized  software  organizations.  Neither  is  size  alone  directly  related  to 
organizational  performance  in  a  consistent  manner.^^ 

As  mentioned  in  Section  3.3,  the  relationships  we  found  between  maturity  level  and 
organizational  performance  persist  among  organizations  of  varying  size.  All  of  the 
relationships  controlling  for  size  approach  statistical  significance  (at  the  .05  level  according  to 
chi-square  criteria).  Among  the  smaller  organizations  with  70  or  fewer  software  employees, 
there  are  significant  correlations  between  maturity  level  and  two  performance  factors  (ability 
to  meet  schedule  and  staff  morale  /  job  satisfaction).  There  are  also  two  statistically  significant 
relationships  (product  quality  and  staff  productivity)  among  the  larger  organizations  that 
employ  over  200  software  personnel.  Four  of  the  relationships  are  significant  among  the 
reportedly  mid-sized  organizations  (all  but  customer  satisfaction  and  staff  productivity). 

Organizational  size  alone  is  unrelated  to  overall  SPI  success  (as  discussed  in  Section  5). 
Respondents  who  represent  larger  organizations  are  no  more,  or  less,  likely  to  claim  such 
success  than  are  those  who  come  from  smaller  organizations. 

In  fact,  we  are  able  to  find  very  few  characteristic  differences  that  are  directly  attributable  to 
organizational  size.  However,  this  lack  of  variation  due  to  size  does  speak  to  the  on-going 
debate.  We  fail  to  find  differences  that  some  might  expect.  In  particular,  respondents  from 
small  organizations  are  no  more,  or  less,  likely  than  those  from  large  organizations  to  complain 
that  software  processes  have  become  more  rigid  and  bureaucratic  since  their  organizations 
embarked  on  their  CMM-based  process  improvement  efforts.  Neither  do  the  respondents 
from  the  varying  sized  organizations  differ  in  their  likelihood  of  saying  that  their  appraisals 
and/or  the  CMM  have  led  to  neglect  of  other  important  issues  facing  their  organizations.  They 
are  equally  likely  to  say  that  their  appraisals  represented  money  well  spent,  and  that  the 
appraisals  had  a  major  positive  effect  on  their  organizations. 

If  anything,  our  survey  data  provide  some  limited  evidence  that  the  smaller  organizations  may 
be  more  amenable  to  successful  process  improvement  than  are  the  larger  ones.  After  all,  it 
typically  is  difficult  to  accomplish  change  in  large  organizations  in  general. 


Staff  productivity  may  be  an  exception.  Sixty-nine  percent  of  the  respondents  from  organizations  with  70  or 
fewer  software  employees  say  that  their  staff  productivity  is  excellent  or  good.  The  comparable  figure  for 
organizations  with  over  200  software  employees  is  45  percent. 
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Our  respondents  from  smaller  organizations  are  less  likely  to  report  unrealistic  expectations 
about  the  cost  or  time  necessary  to  accomplish  tangible  process  improvement.  For  example, 
73  percent  of  those  from  organizations  with  over  200  software  employees  agree  that  the  effort 
is  costing  more  than  they  expected;  47  percent  of  those  from  organizations  with  70  or  fewer 
software  employees  say  the  same.  Similarly,  80  percent  of  those  from  the  largest 
organizations  say  that  the  effort  is  taking  more  time  than  they  expected;  62  of  those  from  the 
smaller  organizations  agree. 

Perhaps  not  surprisingly,  organizational  politics  and  turf  guarding  are  more  difficult  problems 
in  the  larger  software  organizations.  Turf  guarding  is  identified  as  a  substantial  issue  by  32 
percent  of  those  from  the  large  organizations  in  our  sample;  only  7  percent  of  those  from  the 
smaller  organizations  report  a  similar  situation.  Somewhat  similarly,  45  percent  of  those  from 
the  large  organizations  report  a  substantial  amount  of  organizational  politics;  20  percent  of 
those  from  smaller  organizations  do  so. 

Again  not  surprisingly,  more  paperwork  is  necessary  to  get  things  done  in  the  larger 
organizations.  Of  those  from  organizations  with  over  200  software  employees,  70  percent  say 
that  a  substantial  or  moderate  amount  of  paperwork  is  required;  the  comparable  number  is 
47  percent  for  those  who  come  from  organizations  with  70  or  fewer  software  employees. 

Finally,  those  from  smaller  organizations  are  somewhat  less  likely  to  report  that  there  is  a 
common  feeling  among  their  technical  staffs  that  process  improvement  gets  in  the  way  of  their 
“real”  work.  Of  those  from  organizations  with  70  or  fewer  software  employees,  33  percent  say 
that  SPI  is  perceived  to  be  “in  the  way”  at  least  moderately  often;  the  comparable  number  is 
57  percent  for  those  who  come  from  organizations  with  over  200  software  employees. 
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Appendix  C  Predictors  of  Process  improvement 

C.1  Possible  Barriers  and  Success  Factors 

Adequate  commitment  to  process  improvement,  and  the  existence  of  sufficient  resources,  are 
often  thought  to  be  crucial  for  ensuring  successful  SPI  efforts.  As  seen  in  Figures  C-1  and 
C-2,  our  respondents  report  sometimes  substantial  differences  in  commitment  and  resources 
for  SPI  In  their  software  organizations. 


%  “substantial” 
or  “moderate”... 


*  %  “excellent”  or  “good’ 


tangible  incentives  &  rewards  for  SPI 
staff  time  /  resources  dedicated  to  SPI 
sr.  mgt.  understands  technical  issues 
SPI  goals  well  understood 
sr.  mgt.  monitors  SPI  progress 


Figure  C-1 :  Commitment  and  Resources:  Possible  Success  Factors 


%  “substantial” 
or  “moderate”... 


decreasing  demand  for  products  &  services 
middle  management  turnover 
technical  staff  turnover 
senior  management  turnover 
discouragement  about  SPI  prospects 
reorganization  /  staff  downsizing 
SPI  gets  in  the  way  of  “real"  work 


0  25  50  75  100% 


Figure  C-2:  Commitment  and  Resources:  Possible  Barriers 


Similarly,  management  style  and  organizational  cultural  differences  are  often  cited  as 
important  determinants  of  success  or  failure  in  software  process  improvement.  Variations  in 
the  respondents’  reports  about  a  set  of  such  factors  are  summarized  in  Figures  C-4  and  C-3. 
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Figure  C-3:  Management  Style  And  Organizational  Culture:  Possible  Barriers 


%  “substantial” 
or  “moderate”... 


clear,  compensated  SPI  assignments 
management  willing  to  take  risk 
technical  staff  involved  in  SPI 
SPI  people  well  respected 


0  25  50  75  100% 

Figure  C-4:  Management  Style  And  Organizational  Culture:  Possible  Success  Factors 


Other  factors  might  also  affect  the  likelihood  of  success  in  a  process  improvement  effort. 
Among  those  we  discussed  in  Section  6  are  the  scope  of  the  findings  and  recommendations 
raised  in  the  appraisal,  and  the  need  for  guidance,  mentoring,  or  assistance  in  implementing 
the  improvements  suggested  by  the  appraisal. 

We  correlated  each  of  these  factors  with  the  question  (summarized  in  Figure  5-3  on  page  20) 
about  overall  success  in  addressing  the  findings  and  recommendations  of  the  appraisal.  The 
different  factors  distinguish  among  more  and  less  successful  SPI  efforts  in  several  interesting 
ways. 
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C.2  Impact  on  Process  Improvement 

Following  are  the  figures  not  included  in  Section  6  that  show  statistically  significant 
relationships  (p  <  .05  by  chi-square  criteria)  with  reported  success  in  addressing  appraisal 
findings/recommendations. 


well  respected  SPI  personnel 
%  “substantial”  or  “moderate” 


Success  addressing  findings  /  recommendations 


Figure  C-5:  Successful  SPI  and  Well  Respected  SPI  Personnel 


technical  staff  involved  in  SPI 
%  “substantial”  or  “moderate” 


little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 

Figure  C-6:  Successful  SPI  and  Technical  Staff  Involved  In  SPI 
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staff/resources  dedicated  to  SPI 
%  “good”  or  “excellent” 

100-1 - 


Success  addressing  findings  /  recommendations 

Figure  C-7:  Successful  SPI  and  Staff/Resources  Dedicated  to  SPI 


clear  SPI  goals 
%  “substantial”  or  “moderate” 

100-1 - 


little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 

Figure  C-8:  Successful  SPI  and  Clear  SPI  Goals 


“turf  guarding” 

%  “substantial”  or  “moderate” 


little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 


Figure  C-9:  Successful  SPI  and  “Turf  Guarding” 
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previous  discouragement  or  cynicism 
%  “substantial”  or  “moderate” 

100- - - 

75- - - - 


little,  if  any  limited  moderate  substantial  marked 

Success  addressing  findings  /  recommendations 

Figure  C-10:  Successful  SPI  and  Previous  Discouragement  or  Cynicism 


SPI  gets  in  the  way  of  “real”  work 
%  “substantial”  or  “moderate” 


Success  addressing  findings  /  recommendations 

Figure  C-11:  Successful  SPI  and  SPI  Gets  in  the  Way  of  “Real”  Work 


need  guidance  about  how  to  improve 
%  “substantial”  or  “moderate” 


little,  if  any  limited  moderate  substantial  marked 


Success  addressing  findings  /  recommendations 


Figure  C-12:  Successful  SPI  and  Need  Guidance  About  Howto  Improve 
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need  mentoring/assistance 
%  “agree”  or  “strongly  agree” 


little,  if  any  limited  moderate  substantiai  marked 


Success  addressing  findings  /  recommendations 

Figure  C-13:  Successful  SPI  and  Need  for  Mentoring/Assistance 


As  noted  in  the  footnote  on  page  25,  the  data  in  Figure  C-13  are  “noisy”:  there  is  in  fact  a 
statistically  significant  relationship  between  the  respondents’  answers  to  the  two  questions, 
but  it  is  masked  by  the  way  we  combined  the  answers  for  the  full  series  of  bar  graphs.  As  seen 
in  Figure  C-14,  more  characteristic  differences  exist  for  the  respondents  who  either  strongly 
agree  or  strongly  disagree  that  there  is  a  need  for  additional  mentoring  and  assistance  in  their 
organizations.^"^ 


%  needing  more  mentoring/assistance: 

UH  strongly  agree  H  strongly  disagree 
50 

40 

30 

20 

10 

0 

little,  if  any  limited  moderate  substantial  marked 
Success  addressing  findings  /  recommendations 

Figure  C-14:  Successful  SPI  and  Varying  Need  for  Mentoring/Assistance 


There  is  an  unexplained  upturn  among  those  who  say  they  have  had  marked  success  in  addressing  the 
findings  and  recommendations  of  their  appraisals  and  agree  strongly  that  there  is  a  need  for  more  mentoring 
and  assistance,  but  there  are  too  few  respondents  to  tell  whether  or  not  it  is  due  simply  to  chance. 
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Appendix  D  The  Questionnaire 

Th©  survey  questions  focus  on  the  value  and  accuracy  of  the  appraisal,  and  the  success  with 
which  the  findings  and  recommendations  of  the  appraisal  have  been  addressed.  Of  particular 
interest  to  the  process  improvement  community,  we  included  a  number  of  questions  about 
organizational  performance,  e.g.,  product  quality  and  ability  to  meet  schedules.  For 
comparative  purposes,  we  also  asked  a  series  of  questions  about  the  organization  that  was 
appraised. 

Ideas  for  the  content  of  the  questionnaire  came  from  several  sources.  These  include  feedback 
from  informal  “birds  of  a  feather”  sessions  at  the  1994  SEPG  National  Meeting  in  Dallas  and 
the  1994  Software  Engineering  Symposium  in  Pittsburgh.  We  also  reviewed  published  and 
unpublished  critiques  of  the  CMM  and  process  appraisal  methods,  and  sought  out  the  views 
and  concerns  of  various  SEI  stakeholders.  We  pre-tested  an  initial  draft  of  the  questionnaire 
with  SEI  resident  affiliates  (experienced  software  developers  and  managers)  in  November 
1994. 

Most  of  the  questions  are  phrased  in  a  closed-ended  manner.  That  is,  we  pose  a  question  to 
the  respondents  and  provide  a  series  of  pre-coded  answers  from  which  they  are  asked  to 
choose.  Other  questions  ask  for  the  respondents  to  supply  numerical  estimates.  In  general, 
we  avoided  open-ended  questions  in  which  respondents  are  asked  to  provide  free  form, 
textual  answers.^® 

All  survey  data  rely  on  the  self-reports  of  their  respondents,  and  we  cannot  know  with  certainty 
on  what  they  base  their  answers.  However,  survey  data  are  necessary  if  we  wish  to 
generalize  beyond  a  few  selected  instances.  Well  defined  process  and  performance  metrics 
still  are  uncommon,  especially  in  level  1  organizations,  and  metrics  based  on  the  same, 
shared  definitions  are  not  widely  available  for  different  organizations.  Regular  recording  of 
information  does  not  ensure  its  accuracy  in  any  event,  even  in  existing  metrics  programs. 

Surveys  can  ask  for  factual  information  as  well  as  opinions,  and  there  is  evidence  that  people 
do  try  to  answer  survey  questions  honestly.  For  example,  self  reported  and  appraised  maturity 
levels  are  quite  consistent  in  this  survey  (see  Section  3.1),  People  also  differ  substantially  in 
their  descriptions  of  process  improvement  in  their  organizations,  and  those  descriptions  vary 
among  each  other  in  understandable  ways. 

A  facsimile  of  the  questionnaire  on  which  this  report  is  based  is  reproduced  on  the  following 
pages. 


It  is  difficult  to  phrase  good  open-ended  questions,  especially  in  a  self-administered  survey  without  the 
intervention  of  a  skilled  interviewer.  Hence,  it  is  difficult  to  elicit  comparable  results.  Providing  meaningful 
answers  is  difficult  for  the  respondents,  who  tend  to  write  little. 
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Software  Process: 
Assessments  and  Improvement 


This  document  contains  questions  about  a  software  process  assessment  that  was  held  recently  at  your  organization. 
The  questions  ask  about  the  conduct  and  results  of  the  assessment,  and  about  what  has  happened  to  process 
improvement  in  your  organization  since  the  assessment 

This  is  the  first  comprehensive  survey  ever  done  of  assessed  organizations.  The  results  will  be  used  to  provide 
guidance  for  future  SEl  process  improvement  efforts.  The  results  also  will  allow  you  to  make  valuable  comparisons 
among  organizations  similar  to  your  own.  Of  course  you  will  receive  a  summary  of  the  survey  results. 

Your  answers  are  very  important  You  are  part  of  a  carefully  chosen  sample.  It  is  very  important  that  you  return  your 
completed  questionnaire  in  order  for  the  results  to  be  accurate  and  useful.  Combined  with  others  from  many  software 
organizations,  your  answers  will  ensure  as  broad  a  description  as  possible  of  experiences  with  software  process 
assessments  and  subsequent  process  improvement 

The  survey  should  take  about  fifteen  or  twenty  minutes  to  complete.  Please  complete  your  questionnaire  right  away, 
while  it*s  still  on  your  mind.  Return  it  to  the  SEI  in  the  enclosol,  postage-paid  envelope. 

Please  read  and  answer  all  of  the  questions.  Feel  free  to  write  in  any  available  space  if  you  wish  to  comment  on  any 
questions  or  qualify  your  answers.  Your  comments  will  be  read  and  taken  into  account 

Your  answers  will  be  held  in  strict  confidence.  Specific  answers  will  not  be  identified  by  organization,  individual,  or 
in  any  other  manner.  Any  information  identifying  you  will  be  used  for  administrative  purposes  within  the  SEI  only. 


Thank  you  for  your  help. 


Software  Engineering  Institute 
Carnegie  Mellon  University 


®  Copyright  1994,  Carnegie  Mellon  University 
This  work  is  sponsored  by  the  U.S.  Department  of  Defense. 


Winter  1994-1995 
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We  will  be  using  optical  scanning  technology,  so  please  print  or  write  neatly. 

•  You  may  use  a  pen  with  dark  ink  or  a  dark  pencil.  Please  do  nai  use  green. 

•  Please  keep  your  marks  within  the  check  boxes.  Any  mark  will  do:  M  E  EQ 

•  Feel  free  to  use  any  available  white  space  if  you  need  extra  room  for  your  written  answers  or  other 
comments,  but  please  don’t  write  over  the  answer  boxes  or  page  border  lines. 


The  Assessment 


In  this  survey,  we  will  be  asking  you  about  the  following  software  process  assessment: 


What  were  the  main  goals  of  the  assessment?  (Please  mark  as  many  as  apply) 

□  PREPARE  FOR  SOFTWARE  CAPABIUTY  EVALUATION  (SCE) 

□  COMPETmVE/MARKETINQ  PRESSURE  TO  DEMONSTRATE  PROCESS  MATURITY 


□  GENERATE  MAfc 
IMPROVEMENT 


;  SUPPORT  AND  BUY-IN  FOR  SOFTWARE  PROCESS 


□  GENERATE  TECHNICAL  STAFF  SUPPORT  AND  BUY-IN  FOR  SOFTWARE  PROCESS 
IMPROVEMENT 

□  ESTABLISH  BEST  PRACTICES  TO  GUIDE  ORGANIZATION  PROCESS  IMPROVEMENT 

□  ESTABLISH  BASELINE  AND/OR  TRACK  THE  ORGANIZATION’S  PROCESS 
IMPROVEMENT 

□  OTHER  (Please  describe  briefly) 


□  DONT  KNOW 
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2  To  the  best  of  your  knowledge,  how  accurately  did  the  assessment  describe  the  organization's  major 
problems  with  software  process?  (Please  mark  one  box) 

□  VERY  ACCURATELY  •  IT  IDENTIREO  ALMOST  ALL  OF  OUR  PROBLEMS 

□  GENERAUY  ACCURATELY  >  IT  MISSED.  OR  MISDIAGNOSED,  A  FEW  PROBLEMS 

□  NOT  VERY  ACCURATELY  -  THERE  WERE  IMPORTANT  ERRORS  OF  OMISSION  AND/ 
OR  COMMISSION 


3  How  well  did  the  assessment  characterize  the  organization's  strong  points?  (Please  mark  one  box) 

□  VERY  WELL  -  IT  GAVE  CREDIT  WHERE  CREDIT  WAS  DUE 

□  REASONABLY  WEU  •  IT  DID  HIQHUGHT  SOME  IMPORTANT  STRENGTHS 

□  NOT  VERY  WELL  -  IT  FOCUSED  ONLY  ON  PROBLEMS  AND  WEAKNESSES 

4  Following  are  several  statements  that  are  sometimes  made  about  software  process  assessments  and  the 
Capability  Maturity  Model  (CMM).  Do  you  agree  or  disagree  with  the  statements? 

(Please  mark  one  box  for  each) 


4. 1  The  assessment’s  findings  and  recommendations  were  too 
ambitious  to  complete  in  a  reasonable  period  of  time . 

42  We  understood  what  needed  to  be  improved,  but  we  needed 
more  guidance  about  how  to  improve  it . 

4.3  The  assessment  results  were  too  dependent  on  the  expertise 

and  judgment  of  the  assessment  team . . . 

4.4  There  wasn't  enough  room  for  the  assessment  team  to  exercise 

its  judgment . . . 

4.5  The  wrong  people  or  projects  were  chosen  to  participate  in  the 

assessment  (be  interviewed,  fill  out  questionnaires,  etc) . 

4.6  There  was  a  lot  of  **ganiing"  -  people  weren’t  fully  honest  with 

the  assessment  team . 

4.7  Reports  and  training  courses  are  all  well  and  good,  but  we  need 

more  individualized  mentoring  and  assistance  to  have  any  real 
hope  of  improving  our  software  process . 

4.&  It  seems  like  we're  always  having  to  take  new  training  courses 
to  keep  up  with  changes  in  SEI  assessment  methods . 

4.9  The  CMM  provides  valuable  direction  about  the  order  in 

which  process  improvements  should  be  made . 

4.10  There  arc  important  areas  that  the  CMM  does  not  address . 

(Please  describe  briefly) 
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Process  Improvement 


1  How  successfully  have  the  findings  and  recommendations  of  the  assessment  been  addressed? 

( Please  mark  one  box) 

□  LITTLE  IF  ANY  APPRECIABLE  SUCCESS  THUS  FAR 

□  LIMITED  SUCCESS 

□  MODERATE  SUCCESS 

□  SUBSTANTIAL  SUCCESS 

□  MARKED  SUCCESS  THROUGHOUT  THE  ORGANIZATION 

□  DONT  KNOW 

2  To  what  extent  has  the  organization’s  software  process  improvement  effort  been  determined  by  the  findings 
and  recommendations  that  were  raised  in  the  assessment?  (Please  mark  one  box) 

□  ALMOST  ALL  OF  IT  IS  BASED  ON  THE  ASSESSMENT  80%) 

□  MOST  OF  IT  IS  BASED  ON  THE  ASSESSMENT  (^50%) 

□  MUCH  OF  IT  IS  BASED  ON  SOURCES  OTHER  THAN  THE  ASSESSMENT  (<  50%) 

□  VERY  LITTLE  IF  ANY  OF  IT  IS  BASED  ON  THE  ASSESSMENT  (<  20%) 

□  THERE  REAUY  HASNT  BEEN  MUCH  OF  AN  EFFORT  TO  SPEAK  OF 

□  DONT  KNOW 

3  Since  the  assessment...  (Please  mark  one  box  for  each) 

3.1  Did  the  organization  that  was  assessed  create  an  action  plan  for  improving  its 

software  process  based  on  the  results  of  the  assessment? . 

3.2  Were  process  action  teams  (PATs)  or  similar  working  groups  established  as  a  result 

of  the  assessment  to  address  specific  process  improvements? . 

3.3  Have  process  changes  been  implemented  in  pilot  qi  demnnstrarinn  projects  based 

on  the  results  of  the  assessment? . 

3.4  Have  process  changes  been  implemented  throughout  the  organization  based  on  the 

results  of  the  assessment? . 


To  the  best  of  your  knowledge,  what  now  is  the  software  process  maturity  level  of  the  organization  that  was 
assessed?  ( Please  mark  one  box) 

□  CMM  LEVEL  1 

□  APPROACHING  LEVEL  2 

□  CMM  LEVEL  2 

□  APPROACHING  LEVEL  3 

□  CMM  LEVEL  3 

□  HIGHER 

□  DONT  KNOW 
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6 


In  your  judgment,  how  much  **buy-in”  and  support  for  process 
(Please  mark  one  box  in  each  column  for  each  question) 


S.l  participants  in  the  assessment?. 
52  organization's  technical  staff? . 

5.3  organization’s  management? ... 

5.4  assessment  sponsor? . 


improvement  has  there  been  among  the... 


Prior  to  the  i  Since  the 


□  □□□  □□□□ 

□  □□□  □□□□ 

□  □□□  □□□□ 

□  □□□  □□□□ 


In  the  organization  that  was  assessed...  (Please  mark  one  box  for  each) 


6.1  How  much  risk  is  management  generally  willing  to  take? . 

6.2  Are  there  tangible  incentives  or  rewards  for  successful  software  process 

improvement? . 

63  How  much  does  '*turf  guarding"  inhibit  the  progress  of  software  process 
improvement? . 

6.4  Is  there  much  organizational  politics? . 

6.5  How  much  paperwork  is  needed  to  get  things  approved? . 

6.6  Does  senior  management  actively  monitor  the  progress  of  software 

process  improvement? . 

6.7  Has  previous  experience  led  to  much  discouragement  or  cynicism  about 

the  prospects  for  successful  process  improvement? . 

6.8  Is  there  a  feeling  among  the  technical  staff  that  process  improvement  gets 

in  the  way  of  their  real  work? . 

6.9  To  what  extent  are  process  improvement  goals  clearly  stated  and  well 

understood? . 

6.10  To  what  extent  does  management  understand  the  issues  faced  by 

practitioners? . 


□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 

□  □  □  □  □ 
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7 

Overall,  how  would  you  characterize  the  progress  of  process  improvement  since  the  assessment? 

{Please  mark  one  box  for  each) 

So  o 

Uj  ^  U/  ^ 

O  ^ 

Co  "T  T 

Q  CO  Q 

o 

7.1 

The  assessment  was  well  worth  the  money  and  effort  we  spent; 
it  had  a  major  positive  effect  on  the  organization . 

□  □ 

□  □ 

□ 

11 

Because  of  the  assessment  (and/or  the  CMM),  we  have 
neglected  other  important  issues  facing  the  organization . 

□  □ 

□  □ 

□ 

13 

Process  improvement  was  overcome  by  events  and  crises; 

other  things  took  priority . 

□  □ 

□  □ 

□ 

7.4 

Process  improvement  has  often  suffered  due  to  time  and 
resource  limitations . 

□  □ 

□  □ 

□  □ 

□  □ 

□ 

□ 

7.5 

Nothing  much  has  changed  since  the  assessment . 

7.6 

Process  change  has  been  easier  than  we  expected . 

□  □ 

□  □ 

□ 

7.7 

The  assessment  was  counter-productive;  things  have  gotten 

worse . . 

□  □ 

□  □ 

□ 

7.8 

Software  processes  have  become  more  rigid  and  bureaucratic; 

it  is  harder  to  find  creative  solutions  to  technical  problems . 

□  □ 

□  □ 

□ 

7.9 

Process  improvement  is  taking  longer  than  we  expected . 

□  □ 

□  □ 

□ 

7.10 

Process  improvement  is  costing  more  than  we  expected . 

□  □ 

□  □ 

□ 

7.11  There  has  been  a  lot  of  disillusionment  over  the  lack  of 

improvement . 

□  □ 

□  □ 

□ 

8 

How  would  you  characterize  the  organization’s...  (Please  mark  one  box  in  each  column  for  each  question) 

Prior  to  the 

Since  the 

appraisal... 

appraisal... 

h-. 

a 

Uj 

w 

1  #  f  # 

a  o  iT 

jf  § 

UJ  o 

8.1 

customer  satisfaction? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.2 

ability  to  meet  budget  commitments? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.3 

ability  to  meet  schedule  commitments? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.4 

product  quality? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.5 

staff  productivity? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.6 

staff  morale  /  job  satisfaction? .  q 

□ 

□ 

□ 

□  □ 

□  □ 

8.7 

staff  time  /  resources  dedicated  to  process  improvement? .  Q 

n 

□ 

□ 

□  □ 

□  □ 
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About  the  Organization 


2 


Does  the  organization  that  was  assessed  still  exist?  (Please  mark  one  box) 

□  YES  •  IN  ESSENTIALLY  THE  SAME  STATE 

□  YES  -  BUT  IT  WAS  SOLO  OR  BOUGHT  OUT 

□  YES  .  BUT  IN  A  REORGANIZED  STATE 

(e.g..wtth  changed  reporting  channela,  or  as  part  of  a  larger  organizational  unit) 


□  NO  •  IT  NO  LONGER  EXISTS  (l.e.,  the  employees  have  been  reassigned  or  fired) 


For  the  remaining  questions  in  this  section:  Please  describe  the  organization  as 
it  existed  at  the  time  of  the  assessment. 


Approximately  how  many  people  are  employed  in  the  organization  that  was  assessed? 

(Please  specify  a  number  for  each  •  Please  do  not  use  commas) 

_  TOTAL  NUMBER  OF  EMPLOYEES 

_  NUMBER  PRIMARILY  ENGAGED  IN  SOFTWARE  DEVELOPMENT  OR 

MAINTENANCE 


In  the  organization  that  was  assessed...  (Please  mark  one  box  for  each  question) 


3.1  Has  there  been  turnover  in  key  senior  management? .  □  □  □  □  □ 

3.2  Has  there  been  involvement  of  technical  staff  in  the  process  improvement 

effort? .  □  □  □  □  □ 

3.3  Have  the  people  who  arc  involved  in  process  improvement  been  respected 
for  their  technical  and  management  knowledge,  and  their  ability  to  get 

things  done? .  □  □  □  □  □ 

3.4  Has  there  been  clear,  compensated  assignment  of  responsibilities  for 

process  improvement? .  □  □  □  □  □ 

3.5  Has  there  been  a  decreasing  demand  for  the  organization’s  products  or  * 

services? .  □  □  □  □  □ 

3.6  Has  there  been  a  major  reorganization(s)  or  staff  down-sizing? .  □  □  □  □  □ 

3.7  How  much  growth  has  there  been  in  staff  size? .  □  □  □  □  □ 

3.8  How  much  turnover  has  there  been  among  middle  management? .  □  □  □  □  □ 

3.9  How  much  turnover  has  there  been  among  the  technical  staff? .  □  □  □  □  □ 
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4 

Approximately  how  many  people  in  the  organization  have  responsibilities  for  software  process 
improvement  and/or  quality  management  -  as  part  of  their  explicitly  assigned  work  efforts? 

(Please  specify  a  number  for  each  •  Please  do  not  use  commas) 

FULL  TIME 

PART  TIME 

CONSULTANTS 

TOTAL  FULL-TIME-EQUIVALENT  PEOPLE 

(full  timers  plus  the  hours  worked  by  part  timers  &  consultants) 

5 

Does  the  organization  that  was  assessed  have  a  software  engineering  process  group  (SEPG),  or  other  unit(s) 
that  performs  similar  functions?  (Please  mark  one  box) 

□ 

YES 

□ 

NO 

6 

Does  the  parent  organization  of  the  organization  that  was  assessed  have  a  software  engineering  process 
group  (SEPG).  or  other  unit(s)  that  performs  similar  functions?  (Please  mark  one  box) 

□ 

YES 

□ 

NO 

□ 

CXDES  NOT  APPLY 

7 

Docs  the  organization  concentrate  its  efforts  on...?  (Please  mark  as  many  as  apply) 

□ 

A  CORE  PRODUCT  LINE  OR  APPLICATION  DOMAIN 
(e.g.,  switches,  guidance  systems,  or  tnformatiorvsystems) 

□ 

A  CORE  TECHNOLOGY 

(e.g.,  distributed  systems,  real-time  embedded  systems,  object-oriented  design,  or  simulators) 

□ 

A  FEW  SIGNIFICANT  CUSTOMERS 

□ 

REUSE  OF  EXISTING  SOFTWARE 

□ 

EXTREMELY  LARGE  OR  COMPLEX  SYSTEMS 

□ 

NEW  OR  POORLY  UNDERSTOOD  DOMAINS  OR  TECHNOLOGY 

□ 

OTHER  SPECIAL  FOCUS  (Please  describe  briefly) 

□ 

NO  SPECIAL  FOCUS 
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How  frequcnily  are  the  organization’s  technical  staff  in  contact  with  the  customers  and  users  of  the  software 
they  develop?  (Please  mark  one  box  in  each  column^ 

Customers  Users 


WEEKLY  OR  DAILY 

MONTHLY . 

LESS  OFTEN . 

RARELY  IF  EVER .. 


□  □ 

□  □ 

□  □ 

□  □ 


8 1  Are  they  in  contact  as  often  as  necessary?  (Please  mark  one  box  in  each  column) 

Customets  Users 


MORE  OFTEN  THAN  NECESSARY 

about  RIGHT . 

LESS  OFTEN  THAN  NECESSARY  . 


□  □ 

□  □ 

□  □ 


How  frequently  do  the  development  projects  have  to  deal  with  changes  in  customer  requirements? 
(Please  mark  one  box) 

□  WEEKLY  OR  DAILY 

□  MONTHLY 

□  LESS  OFTEN 

□  RARELY  IF  EVER 
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